Processing of native and foreign language subtitles in films: An eye tracking study

Foreign language (FL) films with subtitles are becoming increasingly popular, and many European countries use subtitling as a cheaper alternative to dubbing. However, the extent to which people process subtitles under different subtitling conditions remains unclear. In this study, participants watched part of a film under standard (FL soundtrack and native language subtitles), reversed (native language soundtrack and FL subtitles), or intralingual (FL soundtrack and FL subtitles) subtitling conditions while their eye movements were recorded. The results revealed that participants read the subtitles irrespective of the subtitling condition. However, participants exhibited more regular reading of the subtitles when the film soundtrack was in an unknown FL. To investigate the incidental acquisition of FL vocabulary, participants also completed an unexpected auditory vocabulary test. Because the results showed no vocabulary acquisition, the need for more sensitive measures of vocabulary acquisition are discussed. Finally, the reading of the subtitles is discussed in relation to the saliency of subtitles and automatic reading behavior. An increasing number of films are imported from abroad and broadcast in the original foreign language (FL) soundtrack with subtitles added in the native language. This is called standard subtitling, and it is often preferred to dubbing as it is cheaper and keeps the original voice of the actors, thus avoiding the issue of lip synchronicity (Koolstra, Peeters, & Spinhof, 2002). Nowadays, it is often possible to add subtitles in different languages to films or television programs at the press of a button. When watching a film with subtitles, a viewer has to process not only three sources of information (the soundtrack, the subtitles, and the dynamic images in the film), but also the multilingual situation with both FL and native language. Furthermore, the information coming from these different sources may be redundant, which can render the reading of the subtitles less compelling. © Cambridge University Press 2012. The online version of this article is published within an Open Access environment subject to the conditions of the Creative Commons Attribution-NonCommercialShareAlike licence . The written permission of Cambridge University Press must be obtained for commercial re-use. 0142-7164/12 use, available at https:/www.cambridge.org/core/terms. https://doi.org/10.1017/S0142716412000434 Downloaded from https:/www.cambridge.org/core. IP address: 54.191.125.211, on 16 Jan 2017 at 04:10:52, subject to the Cambridge Core terms of Applied Psycholinguistics 35:2 400 Bisson et al.: Native and foreign language subtitles in films Some studies in the 1980s investigated the allocation of attention to the different sources of information in this multimodal situation. They used eye tracking to measure the amount of time a viewer spent looking in the subtitle area as a function of the subtitle’s presentation time with standard subtitles. In an initial study, D’Ydewalle, Muylle, and van Rensbergen (1985) found that participants fixated upon one or two words per subtitle, leading them to conclude that not much reading of the subtitles occurred. However, another study found that participants spent 30% of the subtitle’s presentation time looking in the subtitle area (D’Ydewalle, van Rensbergen, & Pollet, 1987). Unfortunately, the descriptions of these two studies lack many methodological details (e.g., the number of participants, the language background of the participants, and the number of subtitles used in the study). In a study with children from Grades 2 to 6, D’Ydewalle and van Rensbergen (1989) found that the children also spent time looking in the subtitle area, and interestingly, this practice varied depending on the type of film: less time was spent in the subtitle area for an action film. The number of participants in that 1989 study was reported (12 in each experiment); however, the other methodological limitations mentioned above remain problematic. Overall, the first few studies on the processing of subtitles do indicate that viewers spend some of the presentation time looking at the subtitle area. However, because of the limitations of those studies, it is difficult to reach any more definitive conclusions. One of the particular advantages of subtitling over dubbing is the potential for incidental acquisition of FL vocabulary.1 In a study involving 246 primary schoolchildren watching a 15-min film with standard subtitles, Koolstra and Beentjes (1999) found that on a 28-item auditory vocabulary test, scores were higher in the group who had the FL soundtrack and the native language subtitles (20 correct answers) as opposed to the group with the FL soundtrack only (19 correct answers). These results must be treated with caution though because the control group, who were not exposed to the FL, were able to get 18 correct answers on the vocabulary test, and the analysis of variance (ANOVA) was within-subject despite it being a between-subject design. Therefore, it is difficult to conclude that there was vocabulary acquisition. Another study with primary schoolchildren found that scores on 10-item auditory and 10-item written vocabulary tests were higher in the condition with standard subtitles (5 correct answers on the auditory test and 5.8 correct answers on the written test) compared to a condition with both soundtrack and subtitles in the native language (4.1 correct answers on the auditory test, 4.8 correct answers on the written test; D’Ydewalle & van de Poel, 1999). However, this study used still images instead of a dynamic movie and found an advantage for standard subtitles only when the FL (Danish) used in the soundtrack was similar to the participant’s native language (Dutch). Furthermore, the significance values for the post hoc comparisons are not provided in the article. While the results are promising, it would be important to replicate these findings with a longer vocabulary test and a movie with dynamic images before drawing any strong conclusions. The results of both these studies seem to indicate that subtitles are read to some extent and that the soundtrack is processed, thus allowing vocabulary acquisition to occur. However, it is also possible that the participants used the images in the film to form paired associations with the FL words in the soundtrack. It is therefore necessary to ascertain to what extent the use, available at https:/www.cambridge.org/core/terms. https://doi.org/10.1017/S0142716412000434 Downloaded from https:/www.cambridge.org/core. IP address: 54.191.125.211, on 16 Jan 2017 at 04:10:52, subject to the Cambridge Core terms of Applied Psycholinguistics 35:2 401 Bisson et al.: Native and foreign language subtitles in films subtitles are read and to evaluate their usefulness in the process of FL vocabulary acquisition. Another way of using subtitles is to add FL subtitles to a native language soundtrack. This is called reversed subtitling, and it was first investigated in the context of transcribed television and radio programs that did not involve visual images. Although in this case only two sources of information were available, it was concluded that reversed subtitling allowed participants to access meaning through the native language soundtrack and to map this onto the FL in the subtitles (Lambert & Holobow, 1984). In their study using still images, D’Ydewalle and van de Poel (1999) also investigated the acquisition of FL vocabulary using reversed subtitles. Setting aside the limitations mentioned earlier, they found that participants in the reversed subtitling condition (5.1 correct answers) outperformed participants in the control condition (4.1 correct answers) in the 10-item written vocabulary test, which seems to confirm that the FL subtitles were processed. However, a direct measure of the processing of the reversed subtitles using eye tracking would be more informative. A more recent study using eye tracking investigated the reading of standard and reversed subtitles in children and adults using a 15-min animation. D’Ydewalle and De Bruycker (2007) report detailed eye movements of their participants on 114 and 138 subtitles for the standard and reversed subtitling conditions respectively. They found overall that reversed subtitles are skipped more often than standard subtitles (21% compared to 4%), are fixated less (0.59 fixations per word compared to 0.91) and that less time is spent in the subtitle area (26% of the subtitle’s presentation time compared to 41%). Unfortunately, this study (and other prior studies, e.g., D’Ydewalle et al., 1985, 1987; D’Ydewalle & Van Rensbergen, 1989) did not include a control group with no subtitles, and therefore, the above values may reflect some time spent in the subtitle area as a result of other visual aspects of the movie taking place there. However, what seems to emerge from this study is that reading behavior does occur with both standard and reversed subtitles, but there seems to be a preference for native language subtitles. Another method of subtitling presents both the soundtrack and subtitles in the same language, and this is referred to as intralingual subtitling. Originally intended to make films and TV programs accessible to the deaf and hearing-impaired community (Burnham et al., 2008; De Linde & Kay, 1999), intralingual subtitles are also used by language teachers and researchers because they have the potential to help the learner map the phonology to the written words. Written words in intralingual subtitling are not affected by intonation, accents, or background noise; thus, having access to both the soundtrack and subtitles allows for easier word segmentation by indicating which words are being spoken (Bird & Williams, 2002; Mitterer & McQueen, 2009). One study investigated the processing of intralingual native language subtitles while participants watched either a 12-min English or 20-min Dutch film (D’Ydewalle, Praet, Verfaillie, & van Rensbergen, 1991). The results showed that even though the native language subtitles were not necessary for the comprehension of the film, as a native language soundtrack was available, participants still read them approximately 20% of the time. The authors argued that participants read the subtitles because it was more efficient than listening to the soundtrack. However, as the same message was available in both soundtrack use, available at https:/www.cambridge.org/core/terms. https://doi.org/10.1017/S0142716412000434 Downloaded from https:/www.cambridge.org/core. IP address: 54.191.125.211, on 16 Jan 2017 at 04:10:52, subject to the Cambridge Core terms of Applied Psycholinguistics 35:2 402 Bisson et al.: Native and foreign language subtitles in films and subtitles, it is highly possible that participants followed along with the written text while they heard the message aurally. Many vocabulary-learning studies seem to confirm that having both written and aural form of a word facilitates learning, which would imply that both are processed (Hu, 2008; Ricketts, Bishop, & Nation, 2009; Rosenthal & Ehri, 2008). It is crucial to note that these studies did not include dynamic images. Although studies of FL films with intralingual subtitles seem to show promising results in terms of improvement in speech performance (Borras & Lafayette, 1994), speech perception (Mitterer & McQueen, 2009), and word form-meaning associations (Sydorenko, 2010), indicating that both phonological and orthographic forms are processed, it is still necessary to investigate the reading of the subtitles when all three sources of information (FL soundtrack, FL subtitles, and dynamic images) are present. Overall, the results of prior studies of incidental acquisition of FL vocabulary through watching films with subtitles provide some indirect evidence that the subtitles are processed. In addition, some studies have used eye tracking to directly measure the processing of the subtitles, but those studies are hampered by methodological issues or a lack of crucial information that makes them unreplicable. The aim of the current study was to use a film with standard, reversed, or intralingual subtitles to clarify the processing of the subtitles in each condition. Based on the previous studies mentioned above, it was predicted that participants would read the subtitles regardless of the subtitling condition, but that the reading behavior, as assessed by the number and duration of the fixations, would differ in each subtitling condition. More specifically, the duration and number of fixations should be higher in the standard condition as the subtitles are in the native language, followed by the intralingual condition where both subtitles and soundtrack are in a FL, and finally the reversed condition where the FL subtitles are superfluous and the FL is unknown to the participants. Although the main focus of the experiment was on the reading behavior, an auditory vocabulary test was used to assess the incidental acquisition of FL vocabulary in each subtitling condition. Dutch was chosen as the FL in the film because most people in the United Kingdom have no knowledge of it. However, because Dutch comes from the same Germanic language family as English, the two languages have many similarities, which could aid the acquisition of Dutch vocabulary.